3-dimensional data processing apparatus and method

ABSTRACT

Provided are three-dimensional (3D) data processing apparatus and method. 3D stereoscopic image common format data preserving common 3D image information is defined by converting 3D stereoscopic image data of each format to apply the same 3D image processing algorithm to 3D data of various formats, and native format data of each format is maintained to maintain merits of each format, and a format-specific algorithm table is maintained with respect to a common 3D image processing function, whereby the same 3D image processing algorithm may be applied to various 3D data and a unique algorithm allowing for maintaining the merits of each 3D data may be applied.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority under 35 U.S.C. §119 to Korean PatentApplication No. 10-2014-0006675, filed on Jan. 20, 2014, the disclosureof which is incorporated herein by reference in its entirety.

TECHNICAL FIELD

The present invention relates to an apparatus and method for processingthree-dimensional (3D) data obtained through various sensors, and moreparticularly, to a 3D data processing apparatus and method for applyinga 3D stereoscopic image processing algorithm to various types of 3Dstereoscopic image data.

BACKGROUND

With the advancement of sensors for recording data having 3D informationsuch as laser scanning, a stereoscopic camera, a structured lightscanner, the Microsoft Kinect™, a light field camera, and the like,various types of 3D stereoscopic image data may be obtained. Therespective sensors generate 3D data in different formats, and such datain different formats have the merits and demerits, which, thus, arestored in different manners and different algorithms are applied theretoto be processed according to purposes.

A framework of software for 3D stereoscopic image processing performs 3Dstereoscopic image processing using various types of data formats. Sincedata formats are different, when the same stereoscopic image processingalgorithm is applied, data formats are converted into intermediate datain a common format and the stereoscopic image processing algorithm isapplied thereto. Thus, obtained 3D stereoscopic image data is convertedinto intermediate formats and the 3D image processing algorithm issubsequently applied to the converted intermediate format, not to theoriginal obtained format.

However, there may be a native algorithm that may be used only in a dataformat, not a common algorithm, according to types of 3D imageprocessing scheme and types of data. In this case, the native algorithmapplied only to the corresponding format should be used to obtain betterresults in terms of speed and quality.

Also, data storing 3D information since it was obtained or generated(for example, a laser-scanned point cloud, 3D mesh generated throughmodeling by a user, and the like) has a significant difference in datacharacteristics from 3D image data based on a 2D image, so conversion ofsuch data into a common format and application of a common algorithmthereto causes huge data loss.

SUMMARY

Accordingly, the present invention provides a three-dimensional (3D)data processing apparatus and method capable of applying the same 3Dimage processing algorithm to a 3D stereoscopic image data obtained byusing various types of sensors and 3D stereoscopic image data generatedby a designer regardless of data formats, and capable of applying aunique 3D image processing algorithm according to formats to utilize themerits of respective formats.

In one general aspect, a three-dimensional (3D) data processingapparatus includes: a 3D data obtaining unit configured to obtain 3Ddata; an integrated format data generating unit configured to integratenative format data of the 3D data and common format data generated byusing the 3D data to generate integrated format data; and an integratedformat data processing unit configured to process the integrated formatdata according to an image processing algorithm.

The integrated format data processing unit may check whether a nativealgorithm with respect to the native format data of the integratedformat data exists, and when a native algorithm exists, the integratedformat data processing unit may process the native format data of theintegrated format data according to the native algorithm, and when anative algorithm does not exist, the integrated format data processingunit may process the common format data of the integrated format dataaccording to a common algorithm.

When a native algorithm and a common algorithm for processing theintegrated format data do not exist, the integrated format dataprocessing unit may return an error.

The integrated format data processing unit may generate the commonformat data by using 2D image information and 2D depth information ofthe 3D data, and when the 3D data does not include depth information,the integrated format data processing unit may restore depth informationof the 3D data through a depth restoration algorithm, and generatecommon format data.

In another general aspect, a three-dimensional (3D) data processingmethod includes: obtaining 3D data; generating common format data byusing the 3D data; integrating native format data of the 3D data and thecommon format data to generate integrated format data; processing theintegrated format data according to an image processing algorithm; andprocessing the integrated format data according to the image processingalgorithm to output a generated stereoscopic image.

Other features and aspects will be apparent from the following detaileddescription, the drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a structure of athree-dimensional (3D) data processing apparatus according to anembodiment of the present invention.

FIG. 2 is a flow chart illustrating a process of a 3D data processingmethod according to an embodiment of the present invention.

FIG. 3 is a block diagram showing a computer system implementedaccording to an embodiment of the present invention.

DETAILED DESCRIPTION OF EMBODIMENTS

Hereinafter, embodiments of the present invention will be described indetail with reference to the accompanying drawings.

The advantages, features and aspects of the present invention willbecome apparent from the following description of the embodiments withreference to the accompanying drawings, which is set forth hereinafter.The present invention may, however, be embodied in different forms andshould not be construed as limited to the embodiments set forth herein.Rather, these embodiments are provided so that this disclosure will bethorough and complete, and will fully convey the scope of the presentinvention to those skilled in the art.

The terms used herein are for the purpose of describing particularembodiments only and are not intended to be limiting of exampleembodiments. As used herein, the singular forms “a,” “an” and “the” areintended to include the plural forms as well, unless the context clearlyindicates otherwise. It will be further understood that the terms“comprises” and/or “comprising,” when used in this specification,specify the presence of stated features, integers, steps, operations,elements, and/or components, but do not preclude the presence oraddition of one or more other features, integers, steps, operations,elements, components, and/or groups thereof. Hereinafter, embodiments ofthe present invention will be described in detail with reference to theaccompanying drawings.

Terms used in the present invention are defined as follows.

-   -   3D stereoscopic image data: Data that has both 2D color or gray        image information such as a stereo, light field, range scan        data, and the like, and 3D information or that may be extracted        or converted through a unique algorithm    -   3D data: Data that includes 3D stereoscopic image data and that        includes 3D information such as 3D mesh, point cloud, or the        like, not stereoscopic image data, and that may be converted        into an image    -   3D stereoscopic image common format: Format of data converting        3D data to store 2D image information and 2D depth information        required in a common algorithm    -   3D stereoscopic image native format: Format of data as the sum        of original data such as stereo, light field, range scan, and        the like, a state before 3D data is converted into a        stereoscopic image and additional data such as depth data, or        the like, obtained by processing the original data so as to be        easily used.    -   3D stereoscopic image integrated format: Format of data obtained        by adding a 3D stereoscopic image native format to 3D        stereoscopic image common format data    -   3D image processing function, algorithm: Function or algorithm        providing 3D conversion, or the like, such as a 3D special        effect such as a fog impact, or the like, or view conversion by        using a depth value in addition to a color value of a color or        gray image    -   3D depth restoration algorithm: Algorithm of estimating or        calculating a depth of each pixel from 3D stereoscopic image        data not having a depth value of each pixel

FIG. 1 is a block diagram illustrating a structure of a 3D dataprocessing apparatus according to an embodiment of the presentinvention.

The 3D data processing apparatus according to the present inventionincludes a 3D data obtaining unit 100, an integrated format datagenerating unit 110, and an integrated format data processing unit 120.

The 3D data obtaining unit 100 obtains 3D data through a sensor orobtains 3D data generated by software.

The 3D data obtaining unit 100 may obtain a 3D stereoscopic image of anactual subject through a sensor according to types of 3D data, and incase of a subject that does not actually exists or a subject difficultto obtain by a sensor, data virtually created by using software such asa 3D modeling tool by a designer may be obtained as 3D data.

A sensor refers to any type of device capable of obtaining astereoscopic image such as a stereoscopic camera, a light field camera,a laser scanner, and the like, and 3D data includes 3D stereoscopicimage data and data that includes 3D information such as a 3D mesh, apoint cloud, or the like, and that may be converted into an image,besides 3D stereoscopic image data.

The 3D data obtaining unit 100 designates original data of obtained 3Ddata, as native format data or designates data obtained by adding depthinformation to obtained 3D data, as native format data.

Native formats of data output from sensors are determined by types ofsensor to be used. In case of a stereoscopic camera, left and rightcolor images and a depth image calculated by using the left and rightcolor images are a format, and in case of a light field camera, 4D lightfield data and a calculated depth value are a native format. In case ofusing a laser scanner, point cloud data or a 2D color image and a depthimage are a native format.

Unique 3D algorithms are provided by native formats of respective 3Ddata, so types of algorithms provided for each format are defined. Incase of a stereoscopic image, a left/right view conversion function isprovided, and in case of a light field image, an upper/lower left/rightview conversion function is provided.

In a case in which obtained 3D data is data other than 3D stereoscopicimage data, the 3D data obtaining unit 100 may generate imageinformation and depth information by applying rendering.

The 3D data obtaining unit 100 delivers information regarding theobtained 3D data and native format data of the 3D data to the integratedformat data generating unit 110, and in this case, the 3D data obtainingunit 100 may deliver image information and depth information generatedfrom the 3D data together.

The integrated format data generating unit 110 generates integratedformat data by integrating common format data generated by using thenative format data of the 3D data and the 3D data delivered from the 3Ddata obtaining unit 100.

In detail, the integrated format data generating unit 110 generatescommon format data by using 2D image information and 2D depthinformation of the 3D data. In a case that the 3D data includes 2D imageinformation and 2D depth information, the integrated format datagenerating unit 110 may generate common format data by using theinformation included in the 3D data, or otherwise, the integrated formatdata generating unit 110 may generate 2D image information or 2D depthinformation from the 3D data and generate common format data.

Meanwhile, in a case that the 3D data does not include depthinformation, the integrated format data generating unit 110 may restoredepth information of the 3D data through a depth restoration algorithmand generate common format data by using the restored depth information.

The obtained 3D data may include depth data (for example, a laserscanner, or the like) or may not (for example, stereoscopic data, or thelike) according to types of data. Since the common format data requiresdepth data, if data does not have depth data, the integrated format datagenerating unit 110 calculates depth data by applying a 3D depthrestoration algorithm to the data without depth data. When the depthdata is restored, the integrated format data generating unit 110integrates the restored depth data and the 2D image information togenerate data of a 3D stereoscopic image common format, and combines thegenerated data with 3D stereoscopic image native format data to generate3D stereoscopic image integrated data.

After generating the integrated format data by integrating the nativeformat data of the 3D data and the common format data, the integratedformat data generating unit 110 delivers the generated integrated formatdata to the integrated format data processing unit 120.

The integrated format data processing unit 120 processes the integratedformat data delivered from the integrated format data generating unit110 according to an image processing algorithm, and outputs theprocessed stereoscopic image.

In detail, in processing the integrated format data, the integratedformat data processing unit 120 checks whether a unique native algorithmwith respect to the native format data of the integrated format dataexists. When a unique native algorithm exists, the integrated formatdata processing unit 120 processes the native format data by applyingthe native algorithm, rather than using the common format data of theintegrated format data to be processed.

If a native algorithm with respect to the native format data of theintegrated format data does not exist, the integrated format dataprocessing unit 120 processes the common format data of the integratedformat data by applying a common algorithm and outputs a processedstereoscopic image.

Meanwhile, both a native algorithm and a common algorithm with respectto the 3D stereoscopic image processing function to be applied do notexist, the integrated format data processing unit 120 returns an error.

In case of applying a focus changing function as an example of 3Dstereoscopic image processing to a stereo-based 3D stereoscopic imagedata and light field-based 3D stereoscopic image data, when focuschanging is applied to stereoscopic data, since a native algorithm doesnot exist, a focus changing common algorithm is applied to common formatdata to create a result. In case of performing focus changing on lightfield-based data, a light field-dedicated native focus changingalgorithm, without using the common algorithm, is applied to create aresult.

By applying native algorithms by respective formats, better results interms of speed and quality may be obtained, compared to a case ofperforming a common algorithm on common format data causing data loss byprocessing native data once.

FIG. 2 is a flow chart illustrating a process of a 3D data processingmethod according to an embodiment of the present invention.

The 3D data processing apparatus according to the embodiment of thepresent invention obtains 3D data obtained through a sensor or generatedby software in operation S200. When the 3D data is obtained, commonformat data including 2D image information and 2D depth information ofthe 3D data is generated in operation S210, and native format data andcommon format data of the 3D data are integrated to generate integratedformat data in operation S220.

When processing the integrated format data in operation S230, the 3Ddata processing apparatus checks whether a unique native algorithm withrespect to the native format data of the integrated format data existsin operation S240.

When a native algorithm exists, the 3D data processing apparatus appliesthe native algorithm to the native format data of the integrated formatdata to process the same in operation S250, and outputs a processedstereoscopic image in operation S260.

When a native algorithm does not exist, the 3D data processing apparatuschecks whether a common algorithm supporting an image processingfunction to be applied exists in operation S241, and applies the commonalgorithm to the common format data of the integrated format data toprocess the data in operation S251.

If both the native algorithm and the common algorithm with respect tothe image processing function to be applied do not exist, the 3D dataprocessing apparatus returns an error in operation S252.

In the embodiments of the present invention, 3D stereoscopic image dataobtained through various types of sensors may be processed by applyingthe same algorithm and if each data format has different merits, anative algorithm for each format may be applied to utilize the meritsthereof within a single framework.

Also, in the case of applying a common algorithm, since conversion intoa common format has already been performed in advance, a process timetaken for data conversion may be saved in processing a stereoscopicimage.

An embodiment of the present invention may be implemented in a computersystem, e.g., as a computer readable medium. As shown in FIG. 3, acomputer system 300 may include one or more of a processor 301, a memory303, a user input device 306, a user output device 307, and a storage308, each of which communicates through a bus 302. The computer system300 may also include a network interface 309 that is coupled to anetwork 310. The processor 301 may be a central processing unit (CPU) ora semiconductor device that executes processing instructions stored inthe memory 303 and/or the storage 308. The memory 303 and the storage308 may include various forms of volatile or non-volatile storage media.For example, the memory 303 may include a read-only memory (ROM) 304 anda random access memory (RAM) 305.

Accordingly, an embodiment of the invention may be implemented as acomputer implemented method or as a non-transitory computer readablemedium with computer executable instructions stored thereon. In anembodiment, when executed by the processor, the computer readableinstructions may perform a method according to at least one aspect ofthe invention.

A number of exemplary embodiments have been described above.Nevertheless, it will be understood that various modifications may bemade. For example, suitable results may be achieved if the describedtechniques are performed in a different order and/or if components in adescribed system, architecture, device, or circuit are combined in adifferent manner and/or replaced or supplemented by other components ortheir equivalents. Accordingly, other implementations are within thescope of the following claims.

What is claimed is:
 1. A three-dimensional (3D) data processingapparatus comprising: a 3D data obtaining unit configured to obtain 3Ddata; an integrated format data generating unit configured to integratenative format data of the 3D data and common format data generated byusing the 3D data to generate integrated format data; and an integratedformat data processing unit configured to process the integrated formatdata according to an image processing algorithm.
 2. The 3D dataprocessing apparatus of claim 1, wherein the integrated format dataprocessing unit checks whether a native algorithm with respect to thenative format data of the integrated format data exists, and when anative algorithm exists, the integrated format data processing unitprocesses the native format data of the integrated format data accordingto the native algorithm.
 3. The 3D data processing apparatus of claim 2,wherein when a native algorithm with respect to the native format dataof the integrated format data does not exist, the integrated format dataprocessing unit processes the common format data of the integratedformat data according to a common algorithm.
 4. The 3D data processingapparatus of claim 3, wherein when a native algorithm and a commonalgorithm for processing the integrated format data do not exist, theintegrated format data processing unit returns an error.
 5. The 3D dataprocessing apparatus of claim 1, wherein the integrated format dataprocessing unit generates the common format data by using 2D imageinformation and 2D depth information of the 3D data.
 6. The 3D dataprocessing apparatus of claim 1, wherein the integrated format dataprocessing unit converts the 3D data to generate 2D image information or2D depth information, and generates the common format data by using the2D image information or the 2D depth information.
 7. The 3D dataprocessing apparatus of claim 1, wherein when the 3D data does notinclude depth information, the integrated format data processing unitrestores depth information of the 3D data through a depth restorationalgorithm and generates the common format data.
 8. The 3D dataprocessing apparatus of claim 1, wherein the 3D data obtaining unitobtains 3D data through a sensor or by software.
 9. The 3D dataprocessing apparatus of claim 1, wherein the 3D data obtaining unitdesignates original data of the obtained 3D data, as native format dataor designates data obtained by adding depth information to the obtained3D data, as native format data.
 10. The 3D data processing apparatus ofclaim 1, wherein when the obtained 3D data is data other than 3Dstereoscopic image data, the 3D data obtaining unit generates imageinformation and depth information by applying rendering.
 11. Athree-dimensional (3D) data processing method comprising: obtaining 3Ddata by a 3D data processing apparatus; generating common format data byusing the 3D data; integrating native format data of the 3D data and thecommon format data to generate integrated format data; and processingthe integrated format data according to an image processing algorithm.12. The 3D data processing method of claim 11, wherein the processing ofthe integrated format data according to an image processing algorithmcomprises: checking whether a native algorithm with respect to thenative format data of the integrated format data exists; and when anative algorithm exists, processing the native format data of theintegrated format data according to the native algorithm.
 13. The 3Ddata processing method of claim 12, wherein the processing of theintegrated format data according to an image processing algorithmcomprises: when the native algorithm does not exists, processing thecommon format data of the integrated format data according to a commonalgorithm.
 14. The 3D data processing method of claim 13, wherein theprocessing of the integrated format data according to an imageprocessing algorithm comprises: when the native algorithm and the commonalgorithm for processing the integrated format data do not exist,returning an error.
 15. The 3D data processing method of claim 11,wherein, in the generating of the common format data by using the 3Ddata, the common format data is generated by using 2D image informationand 2D depth information of the 3D data.
 16. The 3D data processingmethod of claim 11, wherein, in the generating of the common format databy using the 3D data, the 3D data is converted to generate 2D imageinformation or 2D depth information, and the common format data isgenerated by using the 2D image information or the 2D depth information.17. The 3D data processing method of claim 11, wherein the generating ofthe common format data by using the 3D data comprises: checking whetherthe 3D data includes depth information; and when the 3D data does notinclude depth information, restoring depth information of the 3D datathrough a depth restoration algorithm and generating the common formatdata.
 18. The 3D data processing method of claim 11, wherein, in theobtaining of the 3D data, the 3D data is obtained through a sensor orthe 3D data generated by software is obtained.
 19. The 3D dataprocessing method of claim 11, wherein the obtaining of the 3D datacomprises: when the 3D data is data other than 3D stereoscopic imagedata, generating image information and depth information by applyingrendering.
 20. The 3D data processing method of claim 11, furthercomprising: processing the integrated format data according to the imageprocessing algorithm to output a generated stereoscopic image.